Skip to content

feat(scheduler): add cache_ttl "never" sentinel for always-warm lanes - #245

Merged
ualtinok merged 2 commits into
cortexkit:masterfrom
iceteaSA:cache-ttl-never
Aug 6, 2026
Merged

feat(scheduler): add cache_ttl "never" sentinel for always-warm lanes#245
ualtinok merged 2 commits into
cortexkit:masterfrom
iceteaSA:cache-ttl-never

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

cache_ttl assumes idle > TTL means the provider evicted the prompt cache, so the next prefix rebuild is free. Two consumers act on that assumption:

  1. the scheduler converts a defer pass into an execute pass (scheduler.ts), and
  2. mustMaterialize fires the ttl_idle HARD fold (via hardCacheExpired in transform.ts).

On lanes kept warm by an external keepwarm mechanism (prewarm proxies, dedicated cache-keep tools that re-warm the provider cache out-of-band), the assumption is wrong: the cache never goes cold, MC's lastResponseTime goes stale anyway, and both consumers false-positive. MC then initiates a rebuild it believes is free that is actually a full paid cache-write — measured 450-560K tokens per fold on large sessions. There was no way to express "this lane never goes cold": the config requires a duration.

Fix

cache_ttl: "never" (case-insensitive, works as the string form or any per-model value):

  • parseCacheTtl("never") returns Infinity; both consumers go inert through the existing comparisons (elapsed > Infinity / elapsed >= Infinity are never true) — no new branches in the hot path.
  • The Rust scheduler mirrors the sentinel with u64::MAX (its predicates already use saturating_sub, so no overflow path).
  • Status surfaces render honestly instead of a bogus countdown: /ctx-status shows never expires (always-warm lane); the TUI sidebar gets a JSON-safe cacheNeverExpires flag on StatusDetail (Infinity does not survive JSON-RPC); Pi's status dialog now uses the shared parseCacheTtl (it previously used a private fallback parser that would have shown a 5m countdown).
  • hardCacheExpired is extracted into a pure computeHardCacheExpired helper so the "never" -> Infinity -> false chain has direct unit coverage instead of only flag-consumption coverage.

Docs: CONFIGURATION.md describes the sentinel, what it disables, and the tradeoff — after a genuinely cold start (e.g. the keepwarm process died), the free-fold window is not detected on such lanes; mutations then apply at the execute threshold.

Verification

  • plugin: 2995 pass / 0 fail; pi-plugin: 666 pass / 0 fail; tsc --noEmit clean on both
  • new tests red-verified: reverting the sentinel fails the scheduler tests and the computeHardCacheExpired test
  • Rust: sentinel + predicate + scheduler-decision tests added in scheduler.rs's test module (note: I could not run cargo test locally — the crates workspace has path deps outside this repo; the change is a 3-line early return mirroring the TS logic)
  • schema + generated docs regenerated (build-schema / build-config-docs drift tests green); check:tui-compiled green

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Adds cache_ttl: "never" to disable the idle-TTL heuristic on always-warm lanes, preventing false executes and paid cache rebuilds. Status surfaces now show “never expires” and keep RPC fields JSON-safe; invalid-TTL diagnostics are restored.

  • New Features

    • parseCacheTtl("never") returns Infinity (TS) / u64::MAX (Rust) so idle-execute and ttl_idle never fire; computeHardCacheExpired exported for consistent checks with onInvalid.
    • Status UIs (/ctx-status, TUI, Pi) render “never expires” via a JSON-safe cacheNeverExpires flag; Pi uses the shared parser; docs and schema updated.
  • Bug Fixes

    • Status RPC avoids Infinity by keeping cacheRemainingMs numeric (0) when TTL is “never” and keying on cacheNeverExpires; auto-execute messaging switches to threshold-only.
    • Transform restores invalid-TTL pass outcomes and session logs through the onInvalid callback.

Written for commit 16315e4. Summary will update on new commits.

Review in cubic

Greptile Summary

Adds an always-warm cache sentinel across scheduler and status paths.

  • Parses cache_ttl: "never" as a non-expiring TTL in TypeScript and Rust.
  • Prevents idle-TTL scheduling and hard-fold decisions for always-warm lanes.
  • Propagates JSON-safe non-expiring status semantics through RPC and TUI surfaces.
  • Restores invalid-TTL diagnostics after extracting the expiration helper.
  • Updates tests, schema, and configuration documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain in the fixes associated with the previous review threads.

Important Files Changed

Filename Overview
packages/plugin/src/plugin/rpc-handlers.ts Represents non-expiring cache status with an explicit JSON-safe flag while keeping numeric RPC fields finite.
packages/plugin/src/hooks/magic-context/transform.ts Centralizes hard-cache expiration calculation and preserves the existing invalid-TTL diagnostics.
packages/plugin/src/features/magic-context/scheduler.ts Adds case-insensitive parsing of the never sentinel as positive infinity.
crates/mc-module/src/scheduler.rs Mirrors the non-expiring sentinel in Rust using u64::MAX.
packages/plugin/src/tui/index.tsx Displays non-expiring cache state and threshold-only auto-execution using the RPC flag.
packages/pi-plugin/src/dialogs/status-dialog.ts Uses the shared TTL parser and renders the always-warm state without a countdown.

Reviews (5): Last reviewed commit: "fix(scheduler): keep cacheRemainingMs JS..." | Re-trigger Greptile

Comment thread packages/plugin/src/plugin/rpc-handlers.ts Outdated
Comment thread packages/plugin/src/hooks/magic-context/transform.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 20 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/plugin/src/hooks/magic-context/transform.ts
Comment thread packages/plugin/src/plugin/rpc-handlers.ts Outdated
@iceteaSA

Copy link
Copy Markdown
Contributor Author

Note on the red Check (dashboard): it's pre-existing on master, not introduced here — this PR touches no dashboard files. The same 2 ConfigEditor ⇄ schema parity failures reproduce on a pristine upstream/master checkout at v0.33.0 (48ab531d):

- []
+ [
+   "experimental.mural.enabled",
+   "experimental.mural.model",
+   "fail_closed_blocking",
+   "pi.subagent_extensions",
+ ]

The v0.33.0 schema added those four leaves without updating the dashboard's RENDERED_PREFIXES/OMITTED_BY_DESIGN lists, and experimental.mural.* also trips the "nothing re-introduces the dead experimental.* namespace" test. Happy to send a separate fix PR for the parity lists if useful — the right classification (render vs omit-by-design, and whether mural should live under experimental.) is a maintainer call.

iceteaSA pushed a commit to iceteaSA/magic-context that referenced this pull request Jul 26, 2026
Brings the cache_ttl "never" sentinel (PR cortexkit#245) into the live union
branch so the always-warm lanes can drop the 999h workaround.

Resolutions:
- rpc-handlers.ts: both imports (getSkillMemoryStats + parseCacheTtl).
- rpc-handlers.test.ts / execute-status.test.ts: the PR branch was cut
  before external-memory made buildStatusDetail and executeStatus async,
  so its two new tests called them synchronously and asserted against a
  Promise. Added the awaits; the external-memory describe block also lost
  its closing braces to the merge and was restored.
@alfonso-magic-context

Copy link
Copy Markdown
Collaborator

Implementation looks right on our read (the "never" sentinel threading matches how cache_ttl flows through the scheduler). It needs a rebase onto current master before we can run the review gates — the release waves since it was opened moved the surrounding scheduler code. Happy to review as soon as it's rebased.

@iceteaSA

iceteaSA commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current master (7af5961d) — ready for the review gates.

One conflict, in rpc-handlers.test.ts: master's new buildStatusDetail — storage versions probe describe block landed adjacent to this PR's cacheNeverExpires block. Resolved as take-both; the two describes are independent and both pass.

Gates on the rebased tree:

Gate Result
bun run typecheck (plugin · pi-plugin · cli) clean
plugin bun test 3366 pass / 0 fail
pi-plugin bun test 716 pass / 0 fail
conflict-marker sweep (tree-wide) clean

Note on lint: bun run lint exits 1 on this tree, but the error is pre-existing on master, not from this PR — noUnusedVariables for the unused StoredModelIdRow interface in compartment-chunk-embedding.ts:33, plus two useLiteralKeys warnings in render-mural.test.ts. Those files are untouched by these two commits (git diff upstream/master..HEAD on them is empty) and the unused interface is present on clean master. Left it alone rather than sneaking an unrelated fix into this PR — happy to include it if you'd prefer, or it can go out separately.

Tehan added 2 commits August 6, 2026 09:12
Lanes kept warm by external keepwarm mechanisms (prewarm proxies,
dedicated cache-keep tools) re-warm the provider prompt cache
out-of-band, so MC's idle>TTL heuristic false-positives: both TTL
consumers (scheduler idle-execute and the ttl_idle m[0] fold) initiate
a rebuild believed free that is actually a full paid cache-write
(measured 450-560K tokens on large sessions).

cache_ttl: "never" (string or per-model value, case-insensitive)
disables both consumers: parseCacheTtl returns Infinity, so
elapsed>ttl / elapsed>=ttl never fire. Rust scheduler mirrors the
sentinel with u64::MAX (predicates already use saturating_sub).
Status surfaces render "never expires (always-warm lane)" instead of
a bogus countdown: /ctx-status, the TUI sidebar (JSON-safe
cacheNeverExpires flag on StatusDetail), and Pi's status dialog
(which previously parsed the TTL with a private fallback parser).
hardCacheExpired extracted to a pure computeHardCacheExpired helper
with direct coverage of the never-chain.

Tradeoff documented in CONFIGURATION.md: on a genuinely cold start
the free-fold window is not detected on such lanes; mutations then
apply at the execute threshold.
…TL diagnostics

- rpc-handlers: the cacheNeverExpires branch assigned Infinity to
  cacheRemainingMs; JSON.stringify converts Infinity to null over RPC,
  violating the numeric StatusDetail contract. Use 0 and let the
  cacheNeverExpires flag carry the semantics (the TUI keys on it first).
- computeHardCacheExpired: the extraction dropped the
  invalid-cache-ttl-fallback pass outcome and session log on parse
  failure. Add an onInvalid callback; the transform call site restores
  the exact pre-extraction record + log.
- test: seed last_response_time in the never-TTL status test — the
  guarded branch only runs when lastResponseTime > 0, so the assertion
  was vacuous without it (verified red: Infinity revert now fails it).

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 20 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/plugin/rpc-handlers.ts">

<violation number="1" location="packages/plugin/src/plugin/rpc-handlers.ts:740">
P3: For never-expiring lanes the code collapses both `cacheTtlMs` and `cacheRemainingMs` to `0` as a JSON-safe workaround, relaying correctness entirely to the new `cacheNeverExpires` flag. This is fine for the in-batch TUI consumers (they key on `cacheNeverExpires` first), but the required numeric fields now report values that are indistinguishable from a brand-new/expired lane to any consumer that reads `cacheExpired`/`cacheRemainingMs`/`cacheTtlMs` without checking the flag. Consider documenting this convention on the `StatusDetail` fields (or keeping `cacheTtlMs` at the raw parsed value and only guarding `cacheRemainingMs`) so future RPC consumers don't misinterpret 0 remaining as "expired".</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

detail.cacheTtlMs = safeParseTtl(detail.cacheTtl);
if (detail.cacheTtlMs === Number.POSITIVE_INFINITY) {
detail.cacheNeverExpires = true;
detail.cacheTtlMs = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: For never-expiring lanes the code collapses both cacheTtlMs and cacheRemainingMs to 0 as a JSON-safe workaround, relaying correctness entirely to the new cacheNeverExpires flag. This is fine for the in-batch TUI consumers (they key on cacheNeverExpires first), but the required numeric fields now report values that are indistinguishable from a brand-new/expired lane to any consumer that reads cacheExpired/cacheRemainingMs/cacheTtlMs without checking the flag. Consider documenting this convention on the StatusDetail fields (or keeping cacheTtlMs at the raw parsed value and only guarding cacheRemainingMs) so future RPC consumers don't misinterpret 0 remaining as "expired".

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/plugin/rpc-handlers.ts, line 740:

<comment>For never-expiring lanes the code collapses both `cacheTtlMs` and `cacheRemainingMs` to `0` as a JSON-safe workaround, relaying correctness entirely to the new `cacheNeverExpires` flag. This is fine for the in-batch TUI consumers (they key on `cacheNeverExpires` first), but the required numeric fields now report values that are indistinguishable from a brand-new/expired lane to any consumer that reads `cacheExpired`/`cacheRemainingMs`/`cacheTtlMs` without checking the flag. Consider documenting this convention on the `StatusDetail` fields (or keeping `cacheTtlMs` at the raw parsed value and only guarding `cacheRemainingMs`) so future RPC consumers don't misinterpret 0 remaining as "expired".</comment>

<file context>
@@ -741,11 +734,22 @@ export function buildStatusDetail(
+        detail.cacheTtlMs = safeParseTtl(detail.cacheTtl);
+        if (detail.cacheTtlMs === Number.POSITIVE_INFINITY) {
+            detail.cacheNeverExpires = true;
+            detail.cacheTtlMs = 0;
+        }
         if (detail.lastResponseTime > 0) {
</file context>

@ualtinok
ualtinok merged commit 8e496b8 into cortexkit:master Aug 6, 2026
11 of 14 checks passed
@alfonso-magic-context

Copy link
Copy Markdown
Collaborator

Merged — thanks for the rebase and for the thorough consumer coverage. Reviewed all four TTL consumers (scheduler defer→execute, HARD-fold trigger, /ctx-status, RPC/TUI status) and each handles the sentinel correctly; the RPC test that pins Infinity out of the JSON contract (and seeds last_response_time so the assertion can't pass vacuously) is exactly the kind of test we want. Nice touch replacing Pi's duplicate local TTL parser with the shared one — that removes a drift class along the way. The docs' honest cold-start caveat (a dead keepwarm means no free-fold detection until the execute threshold) is appreciated. Ships in the next release.

ualtinok added a commit that referenced this pull request Aug 6, 2026
Post-merge review on #245 (cubic): collapsing the never-lane to 0 made a
warm lane numerically indistinguishable from a fresh/expired one for any
consumer that reads the numeric fields without the cacheNeverExpires
flag. -1 discriminates by value alone (falsy-value contract: -1 never /
0 expired-or-unset / N live), documented on the wire type. Pi's dialog
is in-process (no JSON boundary) and legitimately keeps Infinity.
@alfonso-magic-context

Copy link
Copy Markdown
Collaborator

Follow-up on cubic's post-merge comment: valid catch, fixed on master (410c1bd). Collapsing the never-lane to 0 made a permanently-warm lane numerically indistinguishable from a fresh or expired one for any RPC consumer that reads cacheTtlMs/cacheRemainingMs without the cacheNeverExpires flag. Both fields now use -1 as the never-expires sentinel — the values discriminate on their own (-1 never / 0 expired / N live), the convention is documented on the StatusDetail wire type, and the flag stays as the readable form. Pi's status dialog keeps Infinity legitimately (in-process, no JSON boundary).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants